Fault tolerance in a supercomputer through dynamic repartitioning

ABSTRACT

A multiprocessor, parallel computer is made tolerant to hardware failures by providing extra groups of redundant standby processors and by designing the system so that these extra groups of processors can be swapped with any group which experiences a hardware failure. This swapping can be under software control, thereby permitting the entire computer to sustain a hardware failure but, after swapping in the standby processors, to still appear to software as a pristine, fully functioning system.

CROSS-REFERENCE

The present invention claims the benefit of commonly-owned, co-pendingU.S. Provisional Patent Application Ser. No. 60/271,124 filed Feb. 24,2001 entitled MASSIVELY PARALLEL SUPERCOMPUTER, the whole contents anddisclosure of which is expressly incorporated by reference herein as iffully set forth herein. This patent application is additionally relatedto the following commonly-owned, co-pending United States PatentApplications filed on even date herewith, the entire contents anddisclosure of each of which is expressly incorporated by referenceherein as if fully set forth herein. PCT patent application US02\05618,for “Class Networking Routing”; PCT patent application US02\05586, for“A Global Tree Network for Computing Structures”; PCT patent applicationUS02\05567, for ‘Global Interrupt and Barrier Networks”; PCT patentapplication US02\05569 for ‘Optimized Scalable Network Switch”; PCTpatent application US02\05618, for “Arithmetic Functions in Torus andTree Networks”; PCT patent application US02\05568, for ‘Data CaptureTechnique for High Speed Signaling”; PCT patent application US02\05587,for ‘Managing Coherence Via Put/Get Windows’; PCT patent applicationUS02\05575, for “Low Latency Memory Access And Synchronization”; PCTpatent application US02\05614, for ‘Twin-Tailed Fail-Over forFileservers Maintaining Full Performance in the Presence of Failure”;PCT patent application US02\05572, for “Fault Isolaton ThroughNo-Overhead Link Level Checksums’; PCT patent application US02\05570,for “Ethernet Addressing Via Physical Location for Massively ParallelSystems”; U.S. patent application Ser. No. 10/258,515, for“Checkpointing Filesystems”; PCT patent application US02\05574, for“Efficient Implementation of Multidimensional Fast Fourier Transform ona Distributed-Memory Parallel Multi-Node Computer”; PCT patentapplication US02\05571, for “A Novel Massively Parallel Supercomputer”;and U.S. patent application Ser. No. 10/083,270, for “Smart Fan Modulesand System”.

This invention was made with Government support under subcontract numberB517552 under prime contract number W-7405-ENG-48 awarded by theDepartment of Energy. The Government has certain rights in thisinvention.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates generally to the provision of faulttolerance in a parallel computer's interconnection networks by softwarecontrolled dynamic repartitioning.

2. Discussion of the Prior Art

A large class of important computations can be performed by massivelyparallel computer systems. Such systems consist of many identicalcompute nodes, each of which typically consist of one or more CPUs,memory, and one or more network interfaces to connect it with othernodes.

The computer described in related U.S. provisional application Ser. No.60/271,124, filed Feb. 24, 2001, for A Massively Parallel Supercomputer,leverages system-on-a-chip (SOC) technology to create a scalablecost-efficient computing system with high throughput. SOC technology hasmade it feasible to build an entire multiprocessor node on a single chipusing libraries of embedded components, including CPU cores withintegrated, first-level caches. Such packaging greatly reduces thecomponent count of a node, allowing for the creation of a reliable,large-scale machine.

SUMMARY OF THE INVENTION

The present invention provides fault tolerance in a supercomputerthrough dynamic repartitioning. A multiprocessor, parallel computer ismade tolerant to hardware failures by providing extra groups ofredundant standby processors and by designing the system so that theseextra groups of processors can be swapped with any group whichexperiences a hardware failure. This swapping can be under softwarecontrol, thereby permitting the entire computer to sustain a hardwarefailure but, after swapping in the standby processors, to still appearto software as a pristine, fully functioning system.

BRIEF DESCRIPTION OF THE DRAWINGS

The foregoing objects and advantages of the present invention for theprovision of fault tolerance in a supercomputer through dynamicrepartitioning may be more readily understood by one skilled in the artwith reference being had to the following detailed description ofseveral embodiments thereof, taken in conjunction with the accompanyingdrawings wherein like elements are designated by identical referencenumerals throughout the several views, and in which:

FIG. 1 illustrates a very simplified 8 node section of a parallelcomputer and the torus links between those 8 nodes. It is a partialillustration of the torus links of a full array of nodes wherein eachnode actually has 6 torus links in + and −x, y, z directions, and thelinks wrap in each logical direction (x, y or z) from the highestnumbered node back to the lowest numbered node, so as to maintain 6torus links in 6 directions for all nodes in the system.

FIG. 2 is a very simplified illustration of a global combining tree ofthe massively parallel supercomputer, and is a partial illustration of afull global combining tree which connects all nodes over an entirepartition of compute nodes.

FIG. 3 illustrates the operation of the link chip which controlsrepartitioning.

FIG. 4 can be viewed conceptually as a floor plan of the massivelyparallel supercomputer and illustrates 9 rows of 8 compute racksseparated by 8 aisles, wherein each of the 8 racks in each row contains2 midplanes, and each midplane contains 8×8×8 compute nodes.

FIG. 5 illustrates the routing of torus signal into and out of a linkcard through cables which connect to neighboring midplanes, through thelink card, and then into and out of the torus on the current midplane.The link ASICs optionally connect the 3-dimensional 8×8×8 torus on thecurrent midplane to the torus of the larger machine.

FIG. 6 illustrates the routing of global tree signals, which are thesignals of the global combining tree network, into and out of a linkcard, though cables which connect to neighboring midplanes, through thelink card, and then into and out of the midplane compute ASICs. The linkASICs and top level compute ASICs collectively determine how the tree onthe current midplane is connected to the global combining tree in thelarger system.

FIG. 7 illustrates the routing of interrupt signals, which are thesignals of the global interrupt signal network, into and out of linkcard through cables which connect the neighboring midplanes, through thelink card ASICs and FPGA and then into and out of the midplane.

DETAILED DESCRIPTION OF THE INVENTION

The massively parallel Supercomputer described in U.S. provisionalapplication Ser. No. 60/271,124 describes a massively parallel computerhaving (x, y, z) (wherein x=64, y=32, z=32) compute nodes connected byseveral separate communication networks. The first of these networks isa three dimensional (3D) torus, in which each compute node connects by 6links in the + and −x, y, z directions to its 6 logically adjacentnearest neighbor compute nodes, and each compute node has 6bidirectional torus ports.

The massively parallel supercomputer comprises 64×32×32 compute nodes,wherein each compute node includes an ASIC with 2 processors, oneprocessor of which performs processing as part of the massively parallelsupercomputer, and the second processor performs message passingoperations.

FIG. 4 can be viewed conceptually as a floor plan of the massivelyparallel supercomputer and illustrates 9 rows of compute racks separatedby 8 aisles to facilitate service. Each of the 9 racks in each row isabout the size of a refrigerator and contains 2 midplanes. Each midplaneis a basic building block and contains 8×8×8 compute nodes, wherein eachcompute node comprises a multiprocessor as explained above.

The physical machine architecture is most closely tied to a 3D torus.This is a simple 3-dimensional nearest neighbor interconnect which is“wrapped” at the edges. All 6 nearest torus neighbors are equallydistant, except for time-of-flight differences such as exist betweendifferent racks of compute node ASICs, making code easy to write andoptimize. Each node therefore supports 6 independent bidirectionalnearest neighbor links.

FIG. 1 illustrates a very simplified view of 8 nodes of a parallelsupercomputer's torus and the links between those 8 nodes, and is apartial illustration of a full array of nodes wherein each node actuallyhas 6 torus links in + and −x, y, z directions. The links wrap in eachlogical direction (x, y or z) from the highest numbered node back to thelowest numbered node, so as to maintain 6 torus links in 6 directionsfor all nodes in the system. FIG. 1 also illustrates schematically an x,y, z coordinate system consistent with the x, y, z coordinate system ofFIG. 4. The massively parallel supercomputer has compute circuit cardsand link circuit cards which plug into the midplane. The circuit cardsare wired in 2×2×2 sub-cubes while midplanes, two per rack, are wired as8×8×8 sub-cubes. The operative 64 k machine is a 64×32×32 torus,although to provide redundancy to compensate for faulty components themachine is physically implemented as a 72×32×32 torus, wherein theadditional 8×32×32 nodes are provided for redundancy purposes to provideextra groups of redundant standby processors.

In addition to the 6 torus links of each node to 6 nearest neighboringnodes, the massively parallel supercomputer includes two othercompletely separate communication link networks. The secondcommunication link network is global combining tree of links, asillustrated in FIGS. 2 and 6. The third communication link network is aset of global interrupt signals, as illustrated in FIG. 7. The combiningtree of links and the global interrupt signals are similar to each otherin their tree structures and provide communication over an entirepartition (64×32×32 compute nodes), of the machine, both of which arediscussed below.

FIG. 2 is a very simplified illustration of a global combining tree ofthe massively parallel supercomputer which extends over the entiremachine, allowing data to be sent from any node to all others(broadcast), or to a subset of nodes. Global sums, minimum and maximumcan also be calculated. Message passing is supported on the globalcombining tree, and controlled by a second processor within each computenode, allowing intensive operations like all-to-all communications toproceed independent of the compute node.

Pursuant to the present invention a multiprocessor parallel computer ismade tolerant to hardware failures by providing extra groups ofredundant standby processors, and by designing the system so that theseextra groups of processors can be swapped with any group whichexperiences a hardware failure. This swapping can be under softwarecontrol, thereby permitting the entire computer to sustain a hardwarefailure but after swapping in the standby processors, to still appear tosoftware as a pristine, fully functioning system.

System Repartitioning

In the massively parallel supercomputer described herein, three mainseparate interconnection networks can benefit from this dynamicrepartitioning: a three dimensional torus, a global combining tree, anda set of global interrupts. The massively parallel supercomputer isorganized into groups of 512 multiprocessors (8×8×8 nodes) per midplane,with link chips that steer signals over cables between midplanes. Thelink chips (6 chips per link circuit card) are the primary way by whichsoftware reconfiguration of the system is enabled.

The massively parallel supercomputer can be logically repartitioned bysoftware control. This permits a large group of racks (as illustrated inFIG. 4), physically cabled together as one system, to be logicallydivided into multiple subsystems. Each of these logically separatedsubsystems can then simultaneously run different code, or some separatedsystems can be serviced while others compute. Logical repartitioningtherefore facilitates code development and system maintenance.

FIG. 3 illustrates, and the following section explains, the operation ofthe link chip which controls repartitioning. The subsequent sectiondetails the types of subdivisions which are possible.

Link Chip

The massively parallel supercomputer's torus, global combining tree andglobal interrupt signals pass through the link chip when trackingbetween different midplanes. This chip serves two functions. First, itredrives signals over the cables between midplanes, improving the highspeed signal shape and amplitude in the middle of a long, lossytrace-cable-trace connection between compute ASICs on differentmidplanes. Second, the link chip can redirect signals between itsdifferent ports. This redirection function is what enables the massivelyparallel supercomputer to be dynamically repartitioned into multiple,logically separate systems.

The link chip performs two types of torus signal redirection for systemrepartitioning, called regular redirection and split redirection.

Regular Redirection

Regular redirection removes one midplane from one logical direction(along either of the x, y or z axes as illustrated in FIG. 4) of thelarge compute system. Regular redirection is shown in Modes 1 and 2 ofFIG. 3. It involves ports C, F, A and B of the link chip. Ports C and Fare attached to the plus direction and minus direction by cables betweenthe current midplane and the higher or lower order midplane in aparticular torus logical direction, x, y or z, as shown at the top ofFIG. 5. These cable connections are show by the arrows in FIG. 4 labeledLogical X cables 40, Logical Y cables 42 and Logical Z cables 44. PortsA and B connect to a midplane torus loop which circles within themidplane through eight compute processors in series, as illustrated inFIG. 3, and also in FIG. 5 as midplane X torus 51, midplane Y torus 52and midplane Z torus 53.

When operating in Mode 1, the link chip routes signals from the previousmidplane through port C, through the current midplane, as illustrated bya midplane torus loop, and on to the next midplane through port F. Itthereby makes the current midplane part of the larger compute system.

When operating in Mode 2, the cable signals from the previous midplaneenter through port C and are passed directly to the next midplanethrough port F, removing the current midplane from the larger computesystem. Also in Mode 2, torus signals on the current midplane areconnected to and loop within the midplane through ports A and B,creating a smaller compute system.

Split Redirection

Split redirection permits dividing a large 64×32×32 node section of themachine into two equal 32×32×32 halves or four 16×32×32 quarters. Asimplemented in the link chip, split redirection could permit a greatvariety of system divisions. However, due to cost and signal integrityconcerns on long cables, split redirection is only physically cabled inthe logical X direction and only on the number of rack rows (FIG. 4)necessary to permit dividing the large system into two equal halves orfour quarters. Split redirection is shown in Modes 3 and 4 of FIG. 3.Eight Modes, 3 through 10, are necessary to accomplish splitredirection, though only two, Modes 3 and 4 are shown in FIG. 3, forpurposes of illustration, and the remaining modes operate in ananalogous manner. In split redirection the link chip redefines the cableparts which it considers to be the plus or minus cable directions toneighboring midplanes. It either redefines the plus direction port fromthe regular port C to split port D or E, or it redefines the minusdirection port from the regular port F to the split port D or E or both.The regular cables are shown by the thin lines with arrows (logical xcables 40, logical y cables 42, and logical z cables 44)in FIG. 4, andthe split cables 46 are shown as fat lines without arrows (near thecenter of logical x cables). The logical x cables extend along the xdirection, and similarly for the logical y cables in the y direction andthe logical z cables in the z direction.

FIG. 4 illustrates how the Logical X cables are connected between racks.The row numbers are indicated by numbers 0–8 on the left. Note that theLogical x cables are often connected to every other row, with cablesbetween rows 0–2, 1–3, 2–4, 3–5, etc. except for the ends with one cable0-1 and one cable 7-8. These cables allow a connection of a midplane toa neighboring midplane along the x axis without any one cable beingunduly long. Similar cable connection schemes can be employed along they and z axes.

The split cables enable x-dimension torus connections other than alongthe regular logical x cables. For instance, if the machine were beingdivided into two smaller machines, with a first machine having rows 0–4and a second machine having rows 5–8, then split cable 46′ could beswitched in place of logical cable 40′, so that the x cables for thefirst machine are now 0-2, 2-4, 4-3, 3-1 and 1-0, and the second machinecould be switched in a similar manner.

Torus Partitioning

FIG. 4 illustrates the massively parallel supercomputer cabling andpartitioning.

Logical repartitioning enables a range of options for how the machinecan be subdivided. FIG. 4 illustrates examples of both regular and splitpartitioning, and shows how a midplane can be isolated from the systemfor service.

Split partitioning can divide the large 72×32×32 cabled massivelyparallel supercomputer into two subsystems of approximately equalhalves, a 40×32×32 subsystem, and a 32×32×32 subsystem. This can be donein one of two ways, to ensure that two 32×32×32 subsystems can always becreated when a midplane is malfunctioning, independent of where thatmidplane is physically located (by using the split cables 46 in themanner as explained above under Split redirection). Either the 40×32×32subsystem is the top five rows of racks and the 32×32×32 subsystem isthe bottom four rows, or the reverse. For example, if a midplane in row1 in the rack indicated by square 4/5 in FIG. 4 needs servicing, then asplit partition can be used to divide the system between the top fourrows of racks and the bottom five rows. In this case the bottom fiverows numbered 0, 1, 2, 3 and 4 form one 40×32×32 subsystem and the topfour rows 5, 6, 7 and 8 (all having racks numbered 6 designing systempartition #6) form a separate 32×32×32 subsystem. Both subsystems can beoperated in these sizes, or they can be further subdivided using regularpartitioning.

Regular partitioning can isolate one 1-midplane (8-node) long sectionfrom any logical torus direction. If a midplane in the 4/5 rack of row 1in FIG. 4 is malfunctioning, then regular partitioning can be used toisolate row 1 in the logical x direction from the rest of the 40×32×32lower system, creating the 32×32×32 system labeled 1 in rows 0, 2, 3 and4 (system #1) and an 8×32×32 system in row 1 whose racks are labeledwith numbers 2, 3 and 4/5. Regular partitioning of this 8×32×32 sectionin row 1 in the logical y direction isolates the 3 and 4/5 racks fromthe 2 racks, giving a 8×24×32 section (2 racks, system #2) and an 8×8×32section (3 and 4/5 racks). Twice regular partitioning of the 8×8×32section in the logical z direction isolates the 4/5 rack and the 3 rack,resulting in an 8×8×16 section (2 rack, system #3) and two 8×8×8sections (4/5 racks, systems #4 and #5), one of which can be servicedwhile all other subdivisions compute. Similar partitioning can be usedin different combinations to subdivide and isolate differentsubsections.

FIG. 5 illustrates the routing of torus signals into and out of a linkcard through cables which connect to neighboring midplanes, through thelink card, and then into and out of the torus on the current midplane.The link ASICs optionally connect the 3-dimensional 8×8×8 torus on thecurrent midplane to the torus of the larger machine. At the top of FIG.5, the + and −x, y, and z signals are coupled to respectively the + and−logical x cables 40, logical y cables 42, and logical z cables 44 ofFIG. 4. The signals to and from “to split 1” and “to split 2” in the xdirection in FIG. 5 are coupled to the + and split cables 46 of FIG. 4.As explained above, the split cables 46 are only provided along the xdirection, although in more complex embodiments they could also beprovided along the y and z directions. The link card includes + and−ASICs for each of the x, y and z directions, which operate as explainedabove with reference to FIG. 3.

Tree and Interrupt Repartitioning

The global combining tree and global interrupt signals are routedthrough the same link chips and cables as the torus signals, as can beseen by comparing the top sections of FIGS. 5, 6 and 7. Regular andsplit repartitioning therefore break the tree into logical subpartitionsin exactly the same way as the torus. Within a logical sub-partition theI/O processors on each midplane are then software reconfigured toconnect the tree within the partition.

FIGS. 6 and 7 illustrate the routing of global combining tree and globalinterrupt signals through cables and link chips (with x, y, z link chipsbeing illustrated) between midplanes which also carry the torus signals.When the link chips are reconfigured, this sets which midplanes areconnected in each of the system's logical partitions. However, uponrepartitioning, the combining tree network and the interrupt signalnetwork both need to be further configured so that the head of thecombining tree and the head of the interrupt signal network are bothdefined throughout each logical machine partition. This can beaccomplished in many ways.

FIG. 6 illustrates the routing of global tree signals, which are thesignals of the global combining tree network, which are routed overprecisely the same cables as the torus signals of FIG. 5.

For the global combining tree, the massively parallel supercomputer usesa group of top-level midplane compute processors (ASICs) on eachmidplane to collectively define which of the six off-midplane cabledirections (signals through link chips) to neighboring midplanes aredefined as up-tree (from a perspective view, towards the top of the treeof FIG. 2), or traveling to a higher logical level in the tree, andwhich are defined as down-tree (from a perspective view, towards thebottom of the tree of FIG. 2). These top level midplane ASICs have threeglobal tree ports each, and the ports can be switched under softwarecontrol to define which ports are up-tree and down-tree. Collectivelythese top level midplane ASICs define one of the six off-midplane cablelinks as up-tree and the other five as down-tree, and they provide atree connection for the other lower level midplane ASICs, as shown inFIG. 6.

FIG. 7 illustrates the routing of interrupt signals, which are thesignals of the global interrupt signal network, which are also routedover precisely the same cables as the torus signals of FIG. 5.

Since the interrupts are simple AND and OR functions, the complexprocessors of the midplane compute ASICs are not required to performarithmetic operations when connecting multiple down-tree branches to anup-tree link. A group of top level compute ASICs is not needed toconnect the interupt network's off-midplane up-tree and down-tree links.The interrupt routing of FIG. 7 can thereby be simplified compared tothe global tree routing of FIG. 6. For the global interrupts the linkchips in FIG. 7 communicate between themselves over lines 54 andtogether present a single bidirectional up-tree signal over lines 55 toa link FPGA (ASICs Floating Point Gate Array) on the link card. ThisFPGA can perform down-tree broadcasts and up-tree AND and OR logicfunctions. It communicates down-tree signals over the five down-treecable connections and into the midplane.

FIGS. 5, 6 and 7 illustrate that the routing of all of the torussignals, the global tree signals, and the interrupt signals between thecables and the link card is precisely the same. All three network travelover the same cables, and each link card handles the routing andrepartitioning of all three types of signals for all three types ofnetworks.

While several embodiments and variations of the present invention for afault tolerance in a supercomputer through dynamic repartitioning aredescribed in detail herein, it should be apparent that the disclosureand teachings of the present invention will suggest many alternativedesigns to those skilled in the art.

1. A method of providing fault tolerance in a parallel computer systemwhich includes a plurality of parallel processors to render the computersystem tolerant to hardware failures comprising: providing the computersystem with extra groups of redundant standby processors, said computersystem comprising an array of a×b×c compute nodes connected as a threedimensional torus wherein each compute node connects by 6 links,including wrap links, in the + and −x, y, z directions to 6 adjacentcompute nodes, and further including communication links over a globalcombining tree of links, and a similar combining tree for a set ofglobal interrupt signals; and, designing the computer system so that theextra groups of redundant standby processors can be switched to operatein place of a group of processors of the computer system whichexperiences a hardware failure, wherein the computer system's torus,global combining tree, and global interrupt signals pass through a linkchip which redirects signals between different ports of the link chip toenable the computer system to be partitioned into multiple, logicallyseparate systems.
 2. The method of claim 1, wherein the switching isunder software control, thereby permitting the entire computer system tosustain a hardware failure, and after switching in of the standbyprocessors, the computer system appears to software as a fullyfunctioning and operative computer system.
 3. The method of claim 1,wherein the computer system comprises a massively parallel computersystem comprising a plurality of substantially identical compute nodes,each of which comprises one or more CPUs, memory, and one or morenetwork interfaces to connect it with other compute nodes.
 4. The methodof claim 1, wherein each compute node includes an ASIC with amultiprocessor, one processor of which performs processing as part ofthe massively parallel supercomputer, and a second processor whichperforms message passing operations of the compute node.
 5. The methodof claim 1, wherein the link chip also serves a second function ofredriving signals over the cables between midplanes.
 6. The method ofclaim 1, wherein the global combining tree and global control signalsare routed through the same link chip and links as signals that arerouted through said three dimensional torus, such that regular and splitredirection and repartitioning change the global combining free of linksinto logical subpartitions in exactly the same way as the threedimensional torus.
 7. The method of claim 6, wherein uponrepartitioning, the global combining tree and interrupt signals arefurther configured so that the head of the combining free and the headof the global interrupt signals are both defined throughout each logicalmachine partition.
 8. A method of providing fault tolerance in aparallel computer system which includes a plurality of parallelprocessors to render the computer system tolerant to hardware failurescomprising: providing the computer system with extra groups of redundantstandby processors, said computer system comprising an array of a×b×ccompute nodes connected as a three dimensional torus wherein eachcompute node connects by 6 links, including wrap links, in the + and −x,y, z directions to 6 adjacent compute nodes, and further includingcommunication links over a global combining tree of links, and a similarcombining tree for a set of global interrupt signals; and, designing thecomputer system so that the extra groups of redundant standby processorscan be switched to operate in place of a group of processors of thecomputer system which experiences a hardware failure, wherein a linkchip performs two types of signal redirection: regular redirection whichremoves one midplane from one logical direction along either of the x,y, or z axes of the computer system, and split redirection which permitsdividing the computer system into two halves or four quarters.
 9. Themethod of claim 8, wherein the switching is under software control,thereby permitting the entire computer system to sustain a hardwarefailure, and after switching in of the standby processors, the computersystem appears to software as a fully functioning and operative computersystem.
 10. The method of claim 8, wherein the computer system comprisesa massively parallel computer system comprising a plurality ofsubstantially identical compute nodes, each of which comprises one ormore CPUs, memory, and one or more network interfaces to connect it withother compute nodes.
 11. The method of claim 8, wherein each computenode includes an ASIC with a multiprocessor, one processor of whichperforms processing as part of the massively parallel supercomputer, anda second processor which performs message passing operations of thecompute node.
 12. The method of claim 8, wherein the global combiningfree and global control signals are routed through the same link chipand links as signals that are routed through said three dimensionaltorus, such that regular and split redirection and repartitioning changethe global combining tree of links into logical subpartitions in exactlythe same way as the three dimensional torus.
 13. The method of claim 12,wherein upon repartitioning, the global combining tree and interruptsignals are further configured so that the head of the combining treeand the head of the global interrupt signals are both defined throughouteach logical machine partition.